…h decode (ATLAS_GDN_DECODE_GRAPH=1)

Collectives were already async (stream-ordered via all_reduce_async). Two speedups:
- register buffers.norm_output() with NCCL so the 16 attention o_proj all-reduces/token use cached IB MR + user-buffer zero-copy (was unregistered).
- ATLAS_GDN_DECODE_GRAPH=1 extends the existing decode CUDA-graph gate (was self.comm.is_none() only) to capture the full tp>1 decode forward (~130 kernels + 64 all-reduces) into one replayable graph; eager fallback on capture failure. Default off; tp=1 byte-identical.

… shard debug logs

The GDN gated-RMSNorm gain is over the value HEAD-DIM ([vd]=128), shared
across all value heads — NOT per-head [nv*vd]. The HeadParallel loader sharded
it on the value-head axis via shard_gdn_value_vector(unit=vd), reading
local_nv*vd*2 = 4096 bytes from the [vd] (~256-512B) buffer -> cuMemcpyDtoDAsync
INVALID_VALUE at model build (both loader sites: FP8-dense + NVFP4). Fix:
replicate norm.weight (every rank keeps the full [vd]); a_log/dt_bias stay
sharded (they ARE per-head [nv]). tp=1 unaffected (slicer was a no-op there).

Also add permanent tracing::debug! (target spark_model::tp_shard / ::concat) to
slice_segments, shard_gdn_value_vector, dense row-parallel, and gpu_concat_rows
so TP shard sizing is inspectable via RUST_LOG=spark_model::tp_shard=debug
instead of ad-hoc eprintln next time.

…ent + graph contiguity

The active decode list arrives in reverse-arrival order ([7,6,..,0] for 8
seqs), so batch position i mapped to SSM pool slot (n-1-i). Both the
batched-recurrent SSM (ssm_batched_recurrent.rs contiguity check:
states[i]==base+i*stride) and the multi-seq CUDA-graph capture (slot==i
assumption) then FAIL → fall back to the eager per-seq serial loop → no
concurrency scaling. The pool assigns consecutive slots {0..n-1}; sorting the
active list ascending by SSM slot (SequenceState::ssm_slot_idx, new pub
accessor) makes position i ↔ slot i, satisfying the contiguity invariant so
the batched paths engage. Reordering the whole ActiveSeq keeps the
position->seq logits mapping consistent; safe for all decode paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoQXbv8a4aVXFPqj8nQqYN
(cherry picked from commit 0b5cf60)

…ruption)

The batched-projection SSM recurrent decode launched conv1d_update_l2norm
once with batch=n on the base deinterleaved pointer. The conv kernel strides
its input by dim(=conv_dim=8192), but deinterleaved is the QKVZ projection
output laid out [Q|K|V|Z] with stride qkvz_size(=12288). So seq b>=1 read from
b*conv_dim instead of b*qkvz_size, pulling the previous seq's Z-gate region
into the GDN scan: correct at n=1, corrupt at n>=2 (measured 0/3 at n=4).

Mirror the proven per-token pattern in trait_decode_batched_conv_gdn.rs: run
the cheap per-channel conv per-seq (batch=1) with pre-offset pointers so the
qkvz_size input stride and conv_dim output stride are both honored. The
expensive GDN scan stays fully batched. Diagnosis via subagent root-cause read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoQXbv8a4aVXFPqj8nQqYN
(cherry picked from commit 632e986)

- Split the GDN HeadParallel slicers out of tp_shard.rs into tp_shard/gdn.rs
  (704 -> 343 LoC; both under the 500 cap) — drops tp_shard.rs from the
  file-size allow-list; allow-list linear_attn_arms.rs (548) + trait_decode_batched.rs (503).
- clippy: repeat().take() -> std::iter::repeat_n() in tp_shard/tests.rs.
- typos: alog_src_off -> a_log_src_off.
- cargo fmt.

After splitting the GDN slicers into tp_shard/gdn.rs, the unit tests (a
sibling module) could no longer reach segment_copy_plan / CopyOp / the
qkv[z]_segments methods (were private, previously visible via super::* when
co-located). Mark them pub(crate) + re-export CopyOp/segment_copy_plan from
tp_shard so tests resolve them via super::*.

…port

The pub(crate) re-export in tp_shard.rs was unused in the lib (non-test)
build (deny(unused) -> error). Import CopyOp/segment_copy_plan directly in
tp_shard/tests.rs from the gdn submodule instead.

slot_idx is already usize on this branch — the `as usize` from the original
cherry-pick trips clippy::unnecessary_cast under deny(clippy::all).

Under pure-TP (--tp-size 2 --ep-size 1: world_size=2, ep_world_size=1) the
MoE experts are REPLICATED (local_expert_range returns all experts when
ep<=1), so every rank computes the full identical MoE output. But the
all-reduce + shared-expert-exclusion gated on comm.world_size()>1, summing
two full outputs -> 2x per MoE layer -> deterministic garbage from the first
token. Gate on config.ep_world_size>1 (the true 'experts are sharded'
invariant) at all 15 sites across 9 forward variants. Single-node/EP/EP+TP
unchanged; only pure-TP flips from wrongly-reducing to correctly-not.

…eckpoints

Load Hcompany Holo-3.1-0.8B/4B/9B (and Ornith dense) at native BF16
instead of the lossy runtime BF16->NVFP4 requant the no-metadata
(Bf16Raw) dense path defaulted to. Adds BF16 arms for the FFN,
attention (Q/K/V/O) and the SSM in_proj_qkvz/out_proj in qwen35_dense.rs,
and gates the batched-decode qkvz path (ssm_batched.rs) so the BF16
build engages the batched dense_gemm instead of dropping to the per-seq
loop. The NVFP4 weights stay installed as the spec-decode/batched fallback.

Fixes a use-after-free that crashed first-layer prefill with CUDA-700:
the FFN overlay installed gate/up/down via dense_auto(), which returns
the WeightStore cached BF16 ptr un-copied, but load_dense_ffn Bf16Raw
arm had already gpu.free() that buffer while building the NVFP4 fallback.
Snapshot fresh D2D copies before the free.

Validated on Holo-3.1-4B: coherent across FI prefill on/off and long
multi-chunk prefill, zero CUDA-700. With --lm-head-dtype bf16 the
agentic multi-turn suite goes 2/3 -> 3/3 vs the NVFP4 head at ~10%
single-stream decode cost (27.8 -> 25.1 tok/s); lossless head is worth
it for these small models (runtime-NVFP4 default was 72% agentic).